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Abstract1 

~  Several  current  theories  of  procedural  knowledge  hypothesize  that  procedures  are  organized 
as  hierarchies  of  goals,  wherein  accomplishing  a  goal  requires  accomplishing  all  or  some  of  its 
subgoals.  This  form  of  knowledge  is  most  naturally  executed  with  the  aid  of  a  temporary  last-in- 
first-out  stack  of  goals.  This  article  presents  evidence  that  a  stack  regime  is  not  flexible  enough  to 
account.for  the  procedural  problem  solving  exhibited  by  a  sample  of  26  third-graders  solving 
subtraction  problems.  Two  alternative  control  regimes  are  investigated.  One  stores  goals  on  an 
agenda  (an  unordered  set)  and  the  other  stores  goals  in  a  tree.  Both  the  agenda  regime  and  the 
tree  regime  employ  a  rule-based  scheduler  that  picks  the  next  goal  for  execution.  Both  regimes 
succeed  at  modelling  our  sublets’  problem  solving  strategies.  The  tree  regime  is  able  to  account 
for  data  from  another  study  as  well  However,  a  closer  examination  of  the  fit  between  models  and 
data  shows  that  some  students  change  their  execution  strategies  in  the  midst  of  problem  solving. 
This  finding  challenges  fundamental  assumptions  underlying  research  on  cognitive  architectures. 
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1.  Introduction 

Much  research  has  been  devoted  to  uncovering  the  cognitive  architecture  that  underlies 
human  problem  solving  and  skill  acquisition.  (See  Newell.  Laird  and  fiosenbloom  (in  press)  for  a 
review  )  Many  models  for  the  cognitive  architecture  have  been  developed,  including  gps  (Ernst  & 
Newell,  1969),  production  systems  (Newell  &  Simon,  1972,  Newell,  1978;  Newell,  1973),  act* 
.Anderson,  1983),  applicative  and-or  graphs  (VanLehn,  1 983a;  VanLehn,  1983b)  and  soar  (Laird, 
Rosenbioom.  &  Newell.  1986).  The  theorists  differ  considerably  on  whether  their  models  are  to  be 
taken  as  literal  models  of  human  cognitive  architecture,  or  as  mere  notations  that  happen  to 
accurately  predict  certain  aspects  of  human  behavior.  Nonetheless,  all  involved  would  agree  that 
people  have  some  kind  of  procedural  knowledge  and  some  mechanism  for  turning  that  knowledge 
into  action  Most  investigators  (but  not  all)  assume  that  the  mechanism  is  fairly  simple,  and  that  its 
translation  of  knowledge  into  action  is  fairly  direct.  That  is,  they  assume  that  the  mechanism  is 
Similar  to  the  mechanisms  used  by  computers  to  execute  computer  programs.  This  report 
classifies  the  various  proposed  knowledge-executing  mechanisms  that  have  been  appeared  in  the 
literature,  and  shows  which  types  of  cognitive  architectures  are  consistent  with  some  new 
e<cerimentai  evidence  from  the  task  domain  of  arithmetic  calculation 

T^e  classification  of  cognitive  architectures  employed  here  is  actually  one  developed  by 
computer  science  as  a  classification  of  program  execution  mechanisms.  The  classification  is 
cased  on  the  features  that  the  mechanism  makes  available  to  the  programmer.  The  classes  are 
caned  ccr-'ci  reg-mes.  Tor  example,  some  programming  languages  ;e  g  ,  lisp  c  pascal;  permit 
'ecw.rs.ve  programs  and  ether  programming  languages  (eg,  =cr"a‘j  basic)  do  not  The 
execution  mechanisms  for  the  recursion-allowing  languages  are  said  to  obey  a  recursive  control 
-eg  me  whue  those  for  =crtran  and  its  ilk  are  said  to  obey  a  ncn-recurs.ve  control  regime 

Control  regimes  are  determined  mostly  by  how  the  machine  stores  'he  temporary  information 
"'at  t  uses  fo  control  the  execution  of  the  prog'am.  so  control  regimes  are  often  named  by  the  type 
:r  data  structure  used  for  'emporary  information  For  instance  the  control  regime  used  by  some 
-ers-ens  of  u-SP  s  cased  on  using  a  iast-m-first-out  stack  for  storing  control  information,  so  it  is 
ca-'ed  a  s'ac*  reg>me  A  stack  regime  s  one  type  of  recursive  control  'egime  Another  type  of 
recursive  control  regime  s  Cased  on  keeping  control  information  m  a  randomly  accessible  list, 
caned  an  agenda,  so  this  control  regime  >s  called  an  agenda  regime  Although  the  nomenclature 


emphasizes  storage  mechanisms,  the  classification  is  defined  by  the  capabilities  it  allows.  For 
instance,  the  agenda  regime  allows  a  type  of  pseudo-parallel  processing  called  co-routining, 
whereas  the  stack  regime  does  not  permit  co-routining. 


Control  regimes  are  classes  of  mechanisms,  so  it  makes  sense  to  ask  what  the  control 
regime  of  the  human  cognitive  architecture  is.  Knowing  the  control  regime  would  tell  us  what 
capabilities  ie  g..  recursion,  co-routinmg)  mental  programs  could  have.  This  in  turn  would  tell  us 
something  about  the  initial  stages  of  skill  acquisition,  where  new  programs  are  "written."  For 
instance,  suppose  we  knew  that  the  human  cognitive  architecture  obeyed  a  stack  regime. 
Because  a  stack  regime  allows  recursive  programs,  we  could  infer  that  people  could  learn 
recursive  mental  programs,  such  as  the  goal  recursion  strategy  for  solving  the  tower  of  Hanoi 
puzzle  Simon.  1975)  or  a  top-down  method  of  coding  lisp  (Anderson,  Farrell,  &  Saurers,  1984). 
On  the  other  hand,  if  the  students'  architecture  obeys  a  non-recursive  control  regime,  then  the 
same  training  would  engender  a  non-recursive  mental  program,  and  in  particular,  the  program 
would  probably  have  separate  pieces  for  each  level  of  recursion  illustrated  in  the  training. 

it  may  seem  that  the  control  regime  of  the  cognitive  architecture  is  so  far  removed  from 
cbsen/ation  that  it  would  be  impossible  to  ascertain  its  identity  experimentally.  However,  it  does 
have  empirical  consequences,  and  they  can  even  be  fairly  direct.  Suppose  the  training  experiment 
just  mentioned  was  performed  for  the  tower  of  Hanoi.  If  the  control  regime  is  nonrecursive,  the 
acquired  program  would  have  separate  pieces  for  each  level  of  recursion.  This  predicts  that  the 
subiect  could  not  solve  problems  requiring  more  recursive  levels  than  the  problems  they  received 
m  framing.  This  is  a  false  prediction  (Anzai  &  Simon,  1979)  People  can  generalize  from  training  on 
small  emblems  to  anger  problems  (e  g  .  from  the  4  disk  version  of  the  tower  of  Hanoi  to  the  6  disk 
version!  Cn  the  other  hand,  if  the  cognitive  architecture  obeys  a  recursive  control  regime,  such  as 
a  stack  regime,  the  acquired  program  could  use  the  same  pieces  of  knowledge  for  all  levels  of 
recursion,  which  accounts  for  how  people  can  transfer  their  competence  from  the  4-disk  to  the 
S-disk  puzz'e  2  This  illustrates  that  the  control  regime  issue  is  not  only  an  important  one.  but  one 
that  makes  testibie  predictions 
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This  article  delineates  four  recursive  control  regimes  (section  i),  presents  an  experiment 
(section  2),  and  shows  that  the  experiment's  results,  when  combined  with  other  data  from  the  same 
task  domain,  are  compatible  with  only  one  of  the  four  control  regimes  (section  3).  However,  when 
a  simulation  model  based  on  this  control  regime  is  fit  to  the  experiment's  data  (in  section  4), 
several  regularities  are  found  that  are  not  consistent  with  the  usual  hypothesis  that  a  subject  has 
just  one  mental  program  for  a  task,  it  seems  instead  that  subjects  acquire  several  strategies  during 
training  and  switch  among  them  during  testing.  The  architectural  implications  of  these  findings  are 
discussed  in  section  5. 

2.  Control  Regimes 

The  four  control  regimes  are  all  recursive  ones.  There  is  convincing  argumentation  that 
procedural  knowledge  is  hierarchical  (Simon.  1969),  and  recursive  control  regimes  are  the  most 
natural  control  regimes  for  such  organizations.  Although  there  is  only  a  little  direct  evidence  for 
recursive  control  regimes  (VanLehn,  19830,  most  current  accounts  of  cognitive  architecture 
assume  some  type  of  recursive  control  regime  (Anderson,  1983;  Laird,  Rosenbloom,  &  Newell, 
1986,  VanLehn,  1983b). 

In  computer  science,  control  regimes  idealize  the  mechanisms  they  describe  because  they 
do  not  mention  the  capacity  limitations  of  the  control  storage,  in  early  lisp,  the  stack  was  limited  to 
ncldmg  a  few  thousand  function  invocations  because  it  was  in  fact  implemented  by  a  table  of  finite 
size  However,  the  control  regime  is  still  called  a  stack  regime,  because  the  last-m-f irst-out 
protocol  is  the  appropriate  characterization  of  its  behavior  in  applying  the  control  regime  idea  to 
cognitive  architectures,  we  will  continue  the  tradition  of  ignoring  capacity  limitations.  In  part,  this  is 
because  the  old  story  of  seven  chunks  of  short-term  memory  has  developed  into  rich,  complicated 
set  of  hypotheses  (see,  e  g.,  Zhang  and  Simon,  1985.  or  Schneider  and  Detweiler,  1987).  For 
nstance,  there  is  evidence  that  massive  training  increases  the  apparent  capacity  for  temporary 
nformation  (Chase  &  Ericsson,  1982).  Clearly,  execution  of  cogmtve  procedures  is  something 
adults  have  had  much  practice  at  Perhaps  this  practice  has  caused  them  to  develop  a  large 
capacity  memory  for  temporary  control  information.  Although  it  is  not  yet  clear  how  one  should 
apply  the  mere  recent  work  on  short  term  memory  to  the  control  component  of  cognitive 
architectures,  it  would  clearly  be  naive  to  assume,  for  example,  that  a  stack  could  hold  at  most 
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seven  goals.3 

Unlimited  capacity  for  control  storage  is  also  the  idealization  employed  by  current  work  on 
architectures  (op  cit.) .  Mostly,  this  idealization  is  left  undefended.  However,  Anderson  (1983) 
explains  that  when  control  information  is  forgotten  due  to  capacity  limitations,  the  person  will 
reconstruct  it  as  needed  from  the  state  of  the  external  world.  Although  no  one  has  tried  to  model 
this  reconstructive  process  or  investigate  it  experimentally,  it  seems  intuitively  plausible.  Indeed,  in 
the  "situated  action"  account  of  procedural  behavior  (Suchman,  1985),  the  whole  notion  of  internal, 
mental  storage  of  control  information  is  replaced  by  a  reconstructive  processes  that  constantly 
interprets  the  external  world  (the  situation)  in  such  a  way  as  to  provide  roughly  the  same 
functionality  as  an  unlimited  capacity  control  store.  Regardless  of  how  control  storage  is 
implemented,  either  as  mental  information  or  interpreted  situations,  the  issue  of  control  regime 
remains.  It  amounts  to  asking  what  kinds  of  information  are  stored  and  what  conventions  govern 
its  access. 

The  four  control  regimes  to  be  investigated  are  delineated  in  subsequent  paragraphs. 

The  deterministic  stack  regime.  Goals  are  accessed  according  to  a  last-in,  first-out 
convention.  Thus,  when  a  goal  calls  a  subgoal,  the  goal's  state  is  "pushed"  onto  a  "stack."  When 
tne  subgoal  is  completed,  the  goal  is  "popped"  from  the  stack  and  resumes  execution.  The  usual 
stack  regimes  fie  .  in  computer  languages  such  as  pascal)  have  the  added  convention  that  the 
order  in  which  subgoals  are  executed  is  fixed.  Every  time  a  goal  is  processed,  its  subgoals  are 
executed  m  the  same  order  Usually,  the  order  in  encoded  by  the  order  in  which  the  subgoals 
appear  in  the  : written)  program.  We  call  this  stack  regime  the  deterministic  stack  regime  because 
subgoal  orders  are  fully  determined.  This  control  regime  is  used  by  Repair  Theory  (Brown  & 
Vanlehn.  1980;  VanLehn.  1983a;  Vanlehn.  1983b).  ACT*  (Anderson.  1983).  GRAPES  (Anderson. 
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Farrell,  &  Saurers,  1984)4  and  other  cognitive  models. 

The  nondeterministic  stack  regime.  This  control  regime  also  accesses  goals  according  to 
the  last-in-first-out  convention.  However,  the  subgoal  order  is  not  fixed,  instead,  there  is  a  distinct 
component  of  the  program  that  is  responsible  for  choosing  which  subgoal  to  execute.  The 
knowledge  encoded  in  this  component  is  called  a  scheduling  strategy  and  the  component  of  the 
architecture  responsible  for  enacting  the  scheduling  strategy  is  called  a  scheduler  in  the 
architecture  literature,  gps  (Ernst  &  Newell,  1969;  Newell  &  Simon,  1972)  was  the  first  to  employ  a 
nondeterministic  stack  regime.  It  obeyed  a  nondeterministic  stack  regime  when  it  was  configured 
to  perform  means-ends  analysis.  When  the  current  goal  has  several  unsatisfied  preconditions,  gps 
treats  these  as  subgcals  and  uses  a  simple  scheduler  to  choose  which  one  to  work  on.  Newell's 
group's  most  recent  general  problem  solver,  scar  (Laird,  Rosenbloom,  &  Newell,  1986)  has  a 
much  more  powerful  scheduler.  However,  scar  still  enforces  the  last-m-first-out  convention  that 
characterizes  means-end  analysis.  All  subgoals  must  be  satisfied  before  a  goal  can  be  popped 
from  the  stack.5 

The  tree  regime.  Architectures  obeying  a  tree  regime  remember  all  goals  ever  invoked 
during  the  course  of  solving  a  problem  and  allow  unrestricted  access  to  all  of  them.  The  goal- 
subgoal  relationships  are  also  stored,  which  means  that  the  stored  information  can  be  viewed  as  a 
tree  of  goals.  As  an  illustration,  suppose  someone  is  following  the  goal  recursion  strategy  for  the 
3-disk  tower  of  Hanoi  .  i  e  .  to  move  a  pyramid  of  N  disks  from  peg  S  to  peg  T,  move  a  pyramid  of 
N-1  disks  from  peg  S  to  0  move  the  Nth  disk  from  S  to  T,  then  move  the  pyramid  of  N-1  disks  from 
3  ’o  T  ,  The  top  goal  m  the  tree  see  table  l)is  "move  the  3  disk  pyramid  from  peg  A  to  peg  C 
Directly  beneath  it  are  the  three  subgoals  (1)  Move  the  2-disk  pyramid  from  A  to  B.  (2)  move  disk  3 
from  A  to  C.  and  (3)  move  the  2-disk  pyramid  from  B  to  C.  Beneath  the  first  subgoal  are  three  more 
goals  ;  1 .1)  move  disk  1  from  A  to  C.  ;  1  2)  move  disk  2  from  A  to  B.  and  :  1  3)  move  disk  i  from  C  to 
3  After  the  person  completes  the  first  move,  subgoal  1  1  is  marked  "satisfied  "  After  the  first  three 
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moves,  goals  1.1.  1.2,  1.3  and  i  are  marked  satisfied.  On  the  fourth  move,  goal  2  is  marked  done, 
and  goal  3,  move  the  2-disk  pyramid  from  B  to  C,  is  expanded  producing  three  subgoals. 
Execution  consists  of  expanding  goals  and  marking  goals  satisfied.  Tree  structure  is  never 
deleted. 

Move  3-disk  pyramid  from  A  to  C 

1 .  Move  2-disk  pyramid  from  A  to  B 

1.1  Move  1 -disk  pyramid  from  A  to  C 

1 .2  Move  disk  2  from  A  to  B 

1 .3  Move  1  disk  pyramid  from  C  to  B 

2.  Move  disk  3  from  A  to  C 

3.  Move  2-disk  pyramid  from  B  to  C 

3.1  Move  1 -disk  pyramid  from  B  to  A 

3.2  Move  disk  2  from  B  to  C 

3.3  Move  1  -disk  pyramid  from  A  to  C 

Table  1 :  Complete  goal  tree  for  3-disk  tower  of  Hanoi  solution 

In  computer  science,  tree  regimes  invariably  employ  a  scheduler  that  is  allowed  to  pick  any 
pending  goal,  where  a  pending  goal  is  leaf  of  the  tree  that  is  not  marked  ''satisfied."  A  scheduling 
strategy  determines  how  the  scheduler  makes  its  choices.  Consequently,  a  tree  regime  can  do 
anything  that  a  nondeterministic  stack  regime  can  do. 

However,  the  tree  regime  permits  a  behavior,  sometimes  called  co-routining  or  time-sharing, 
wherein  control  alternates  back  and  forth  between  two  or  more  tasks.  To  see  how,  suppose  that  a 
tree  has  two  main  goals.  A  and  B,  beneath  its  top  node,  and  that  A  and  B  both  have  numerous 
subgoals.  The  scheduler  can  pick  pending  subgoals  of  A  for  a  while,  then  pick  subgoals  of  B.  then 
go  back  to  choosing  subgoals  of  A.  This  alternation  is  a  form  of  pseudo-parallel  processing.  It  can 
not  be  done  by  a  stack  regime. 

There  have  been  no  experimental  tests  of  whether  people  can  co-routine,  in  part,  this  is  due 
to  the  types  of  tasks  studied  in  the  literature.  Some  tasks,  such  as  the  tower  of  Hanoi,  do  not 
permit  the  usual  recursive  solution  procedures  to  be  executed  in  co-routine  fashion.  Refering  back 
to  the  tree  mentioned  earlier,  the  puzzle  is  constructed  so  that  a  person  physically  cannot  work  on 
goal  3  until  goal  i  has  been  completed. 

in  heu  of  experimental  evidence,  we  can  consult  the  intuition  about  whether  people  can 
co-routme  However,  the  intuition  fails  to  give  a  clear  answer.  For  instance,  an  experienced  cook 


can  alternate  between  chopping  vegetables  for  a  salad  and  basting  a  roast.  At  first  glance,  this 
apparent  co-routining  seems  to  show  that  the  architecture  obeys  a  tree  regime.  However,  because 
we  do  not  Know  what  the  expert  cook's  procedural  knowledge  is,  we  can  not  say  with  certainty  that 
salad-making  and  roast-basting  are  adjacent  goals  in  the  meal  preparation  tree.  Their  subgoals 
could  have  been  combined  into  one  large  salad/roast  goal  while  the  cook  was  learning  how  to 
orchestrate  a  meal.  This  salad/roast  goal  could  be  executed  on  a  stack  regime  architecture, 
yielding  the  same  surface  behavior.  Indeed,  the  circumstances  under  which  beginning  cooks  could 
perform  the  salad/roast  co-routine  are  unknown.  They  might  need  to  use  a  timer,  in  which  case  a 
different,  less  powerful  control  regime  (e  g.,  a  stack  regime  with  the  ability  to  handle  interrupts) 
suffices.  The  moral  of  this  homely  example  is  that  one  can  not  infer  the  control  regime  directly  from 
surface  behavior.  One  must  know  the  structure  of  subjects'  procedural  knowledge.  This  important 
methodological  prerequisite  is  discussed  again  later. 

The  tree  regime  has  the  odd  property  that  satisfied  goals  are  remembered  forever.  This 
might  be  approximately  correct,  if  goal  storage  is  implemented  as  episodic  memory  of  some  kind, 
or  it  could  be  just  an  idealization.  The  last  control  regime  permits  co-routining  but  store  goals  more 
economically. 

The  agenda  regime.  The  agenda  regime  is  like  the  tree  regime,  except  that  only  the  pending 
goals  are  stored.  The  goals  are  viewed  as  an  unordered  set.  As  an  illustration,  consider  again  the 
tower  cf  Hanoi  example.  After  the  first  move,  the  agenda  is  (1.2,  1.3,  2.  3}.  Just  after  the  fourth 
move,  the  agenda  is  (3).  On  the  next  cycle,  the  scheduler  picks  goal  3.  Processing  it  modifies  the 
agenda  to  be  (3.1 .  3.2.  3.3).  Goal  3  has  been  removed,  since  it  is  no  longer  pending,  and  its  three 
subgoals  have  been  placed  on  the  agenda.  An  agenda  control  regime  supports  co-routining  just  at 
the  tree  regime  does. 

Although  agenda  control  regimes  are  common  among  current  Artificial  Intelligence  problem 
solver  ;see  Niit  1 986)  for  a  review  of  a  particularly  popular  one,  called  the  black-board  architecture), 
no  cognitive  architecture  has  employed  one.  it  would  be  an  interesting  direction  to  explore 

This  completes  the  introduction  of  me  tour  control  regimes  to  be  considered  here.  There  are. 
of  course  many  other  control  regimes  in  computer  science.  For  instance,  we  are  ignoring  control 
regimes  for  object-oriented  programming  languages.  We  are  considering  only  control  regimes  for 


von  Neuman  style  architectures  because  those  are  the  ones  that  have  been  employed  successfully 
in  explaining  human  problem  solving  behavior  and  skill  acquisition.  Connectionist  architectures  are, 
so  far,  the  only  challengers,  but  they  have  a  long  ways  to  go  before  they  can  model  behavior  that 
takes  longer  than  a  minute  or  two.  We  are  interested  in  problem  solving  that  takes  several  minutes 
or  hours  to  perform,  so  we  have  concentrated  on  von  Neuman  architectures. 

3.  The  experiment 

Before  discussing  the  experiment  per  se,  some  methodological  issues  will  be  raised  and 
dealt  with. 

3.1.  Methodological  issues 

it  is  difficult  to  unequivocally  determine  which  control  regime  governs  the  cognitive 
architecture  because  one  control  regime  can  emulate  another.  For  instance,  a  nondeterministic 
stack  regime  emulates  a  deterministic  stack  regime  when  the  scheduler  employs  the  scheduling 
strategy  of  ordering  pending  goals  by  their  order  in  the  procedural  knowledge  structure.  Indeed, 
the  four  control  regimes  under  discussion  happen  to  fall  into  a  total  order.  When  listed  in  the  order 
i;  deterministic  stack  regime.  ;2)  nondeterministic  stack  regime,  (3)  agenda  regime,  and  (4)  tree 
regime.  each  control  regime  can  emulate  the  control  regimes  that  precede  it  in  the  list.  The  ability 
cf  one  control  regime  to  emulate  another  means  that  the  determination  of  control  regime  might 
have  to  rely  on  assumptions  of  simplicity  and  parsimony,  if  subjects  are  acting  in  such  a  way  that 
•he  deterministic  stack  regime  can  model  their  behavior,  then  all  the  other  control  regimes  can 
model  their  behavior  as  well  and  one  would  have  to  invoke  parsimony  and  simplicity  in  order  to 
argue  that  they  are  actually  using  a  deterministic  stack  regime  As  it  turns  out.  this  particular 
methodological  difficulty  dees  not  arise,  since  the  data  favor  the  tree  regime,  which  none  of  the 
other  regimes  can  emulate 

A  second  difficulty  m  determining  a  subject's  control  regime  is  that  it  is  not  the  control  regime 
al one  that  determines  the  subject's  problem  solving  behavior.  The  subject  is  executing  some 
Drccedure  or  plan  As  illustrated  earlier  given  an  appropriate  procedure,  even  the  weakest  control 
regime  can  act  just  like  the  strongest.  So  inferences  about  the  subject's  control  regime  are 


impossible  without  some  knowledge  ot  the  procedure  that  they  are  executing. 

in  order  to  make  strong  assumptions  about  the  procedural  knowledge  being  executed,  we 
choose  a  task  domain  in  which  skill  acquisition  is  well  understood.  The  subjects'  task  is  subtraction 
of  multidigit  whole  numbers  (e.g..  324-68).  There  are  several  comments  to  make  about  this  task 
domain. 

Unlike  the  classic  1970's  studies  of  puzzle  solving,  where  subjects  are  given  a  description  of 
the  solution  state  and  asked  to  find  a  path  to  it,  the  subtraction  task  gives  subjects  a  procedure  and 
asks  them  to  follow  it.  However,  these  two  types  of  tasks  are  not  as  different  as  one  might  think. 
Often,  the  subjects  in  puzzle  solving  experiments  invent  partial  plans  and  follow  them.  Structurally, 
plans  and  procedures  are  identical,  and  the  control  regimes  that  can  be  used  in  following  them  are 
the  same  So  the  difference  between  classic  puzzle  solving  and  procedure  following  is  only  in  the 
source  of  the  plan,  procedure  being  followed,  and  not  in  the  way  that  that  procedural  knowledge 
structure  is  followed.  Consequently,  the  claims  presented  here,  supported  by  a  procedure 
following  task,  may  also  hc.d  for  planning  tasks.  Further  research  would,  of  course,  be  required  in 
order  to  test  this  purported  generality. 

Despite  the  fact  that  subjects  are  taught  a  procedure  for  subtraction,  one  can  not  assume 
that  that  specific  procedure  is  the  one  that  they  are  following  because  skill  acquisition  may  not  be 
so  straightforward.  For  instance,  it  is  known  that  some  students  follow  buggy  procedures,  which 
are  systematic  and  stable  procedures  that  happen  to  yield  incorrect  answers  (Brown  &  Burton, 
1973;  VanLehn.  1982).  However,  there  has  been  extensive  work  on  how  subtraction  procedures 
are  acquired.  A  model  exists  that  explains  why  some  students  develop  bugs  (VanLehn.  1983b: 
VanLehn  1983a)  More  importantly  for  the  purposes  of  this  paper,  this  model  generates  a  set  of 
crccedures.  called  cere  procedures,  that  are  potential  outcomes  of  instruction  in  subtraction  Some 
of  the  core  procedures  have  been  observed  (albeit,  indirectly),  and  others  are  predictions  about 
crocedures  that  may  be  obsenved  m  the  future.  We  will  assume  that  the  subjects  in  the  experiment 
'ecorfed  here  are  following  one  of  the  30  core  procedures  generated  by  the  model  when  it  is 
•augnt"  with  the  same  instructional  material  that  the  subjects  were  taught  with  (see 
VanLehnj  1983b),  chapter  2).  This  assumption  replaces  the  simple  (and  false)  assumption  that 
subjects  follow  the  procedure  that  their  teacher  intends  them  to  learn. 
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A  by-product  of  this  assumption  is  that  when  fitting  a  particular  control  regime  to  a  subject's 
behavior,  one  must  choose  a  core  procedure  from  the  set.  Fortunately,  for  the  data  discussed 
below,  there  was  never  any  ambiguity .  only  one  choice  was  appropriate  for  each  subject. 


Another  caveat  to  mention  is  that  the  choices  of  a  procedure  and  a  control  regime  (along 
with  a  scheduling  strategy,  if  the  control  regime  needs  one)  do  not  totally  predict  behavior.  They 
only  predict  behavior  as  long  as  things  go  according  to  plan.  They  do  not  specify  what  happens  at 
impasses,  where  the  procedure/control  regime  says  to  do  something  which  can  not  be  done  given 
the  current  state  of  the  problem.  Impasses  have  been  extensively  studied  (Brown  &  VanLehn, 
1980;  VanLehn,  1983a;  VanLehn,  1983b).  In  situations  where  subjects  may  not  seek  help,  they 
resort  to  one  of  a  small  variety  of  heuristic  repairs,  which  are  local  perturbations  to  the  control  state. 
For  instance,  one  repair  is  deciding  to  give  up  on  the  current  goal.  In  the  stack  regime,  this  is 
implemented  by  popping  the  stack.  In  the  tree  regime,  is  this  implemented  by  marking  the  current 
goal  and  all  its  subgoals  "non-pending."  Our  theory  of  repairs  (called  Repair  Theory)  specifies  this 
set,  although  it  does  not  indicate  the  circumstances  under  which  subjects  will  choose  one  repair  in 
preference  to  the  other  applicable  repairs.  This  means  that  matching  a  model  to  a  subject's 
behavior  requires  fitting  a  parameter,  the  choice  of  repair  at  each  impasse.  Because  repairs 
perform  such  local  changes  to  the  control  state,  this  parameter  does  not  allow  much  control  over 
the  model's  behavior.  (In  particular,  one  can  not  get  one  control  regime  to  emulate  another.)  This  is 
good,  because  it  usually  makes  the  choice  of  repair  at  each  impasse  unambiguous. 

A  last  comment  is  that  subjects  do  not  always  do  what  they  intend.  They  sometimes  make 
unintended  actions,  called  slips  (Norman,  1981).  To  deal  with  slips,  we  edit  them  from  our  data 
fsee  the  discussion  below),  along  with  intended  behavior  that  lies  outside  the  task  domain,  such  as 
rewriting  answers  to  make  them  more  legible. 

To  summarize,  there  are  four  major  difficulties  m  determining  a  subject’s  control  regime:  (1) 
some  control  regimes  can  emulate  others,  so  simplicity  and  parsimony  may  sometimes  play  a  role 
m  deciding  which  regime  the  subiect  has:  (2)  all  control  regimes  execute  a  procedure,  so  one  must 
choose  a  procedure  when  fitting  a  proposed  model  to  a  subject's  behavior;  (3)  the  control  regime 
and  procedure  do  not  determine  the  subject  s  behavior  at  impasses,  so  their  behavior  at  their 
impasses  if  any;  must  be  tit  from  a  set  of  repairs:  t-h  slips  and  othere  behavior  from  outside  the 
task  domain  intest  the  data  None  of  these  difficulties  are  insurmountable,  chiefly  because  of 
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extensive  prior  work  on  the  particular  task  domain  chosen  for  investigation. 

3.2.  Subjects  and  methods 

The  subjects  for  the  experiments  were  drawn  from  three  third-grade  classrooms  The 
classrooms  were  pre-tested  twice  using  a  paper-and-pencil  diagnostic  test.  We  selected  33 
students  whose  errors  on  the  pre-tests  either  showed  an  uncommon  bug  or  were  not  systematic 
enough  to  be  analyzed  as  deriving  from  bugs. 

These  33  subjects  were  tested  individually  in  a  small  room  adjacent  to  their  classroom.  Each 
student  solved  an  individualized  paper-and-pencil  test  whose  items  were  designed  to  elicit  the 
errors  we  saw  on  that  student's  pre-tests.  In  order  to  collect  the  exact  writing  actions,  the  test  page 
was  taped  to  an  electronic  tablet,  and  students  filled  out  the  test  with  a  special  pen.  Equipment 
malfunctions  caused  the  data  from  7  students  to  be  lost.  Tablet  data  from  each  of  the  remaining  26 
students  were  converted  into  a  sequence  of  character-writing  actions,  separated  by  measured 
pauses  5  These  26  sequences  are  the  "protocols"  used  throughout  this  article  as  data. 

3.3.  Results:  qualitative  version 

it  is  difficult  to  summarize  protocol  data  in  a  theory-neutral  way,  so  this  section  will  adopt  a 
classification  that  is  useful  for  the  purpose  of  comparing  control  regimes,  and  summarize  the  data 
in  terms  of  that  classification.  A  later  section  treats  the  data  in  a  more  quantitive  fashion.  The 
observed  orders  are  classified  into  standard  order,  locally  nonstandard  orders,  and  globally 
nonstandard  orders.  A  later  section  will  show  that  all  three  types  of  orders  occurred.  This  section 
merely  defines  the  three  classes,  along  with  some  other  nomenclature  that  will  be  useful  later 

We  define  a  standard  order  to  be  an  execution  order  that  was  taught  to  the  students 
Unfortunately,  we  do  not  know  exactly  which  standard  order  was  taught.  We  know  the  textbooks 
that  the  students  were  taught  from  but  the  textbook's  examples  and  discussions  are  consistent 
with  more  than  one  standard  order  One  would  have  to  watch  the  teacher  present  examples  at  the 


anginal  purpose  of  the  experiment  was  to  find  chronometric  evidence  tor  repair  by  measuring  the  pauses  between 
writing  actions  However,  the  pause  data  turned  out  to  be  very  noisy  Long  pauses  seemed  to  be  caused  mostly  by 
episodes  of  counting  .e  g  .  in  order  to  determine  a  number  fact,  sucn  as  the  difference  between  '5  and  7).  by  ‘iddlmg  with 
—o  sen  or  by  otner  theoretcaify  rreievant  activities  Against  this  high  background  variation  in  pause  length  it  wouid  be 
d  '“'cult  to  measure  the  duration  of  repair  episodes  relative  to  the  duration  of  nonrepair  actions  We  Kave  not  undertaken  a 
■"crough  analyse  of  the  pause  data 
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blackboard  in  order  to  determine  unambiguously  which  orders  were  taught.  However,  under  any  ot 
the  standard  orders  permitted  by  the  textbook,  the  columns  are  processed  m  right-to-left  order,  and 
all  borrowing  actions  for  a  column  are  completed  before  the  answer  to  the  column  is  calculated.  It 
will  be  assumed  that  these  two  properties  define  the  standard  orders  of  subtraction  execution. 

We  found  that  most  sublets  employed  a  standard  order,  but  eight  subjects  ordered  their 
writing  actions  in  nonstandard  ways  ;see  table  2). 7  For  instance,  some  students  did  all  the 
borrowing  first,  on  one  right-to-left  pass  across  the  columns  of  a  problem,  then  returned  left-to-right 
filling  m  the  column  answers.  Although  the  small  sample  size  and  and  its  biased  selection  prevents 
us  from  drawing  strong  conclusions  about  the  overall  population,  the  fact  that  one  third  of  the 
sample  used  nonstandard  orders  indicates  that  phenomenon  is  not  a  trivial  or  idiosyncratic  one. 


Standard  order  15 

Nonstandard  orders  8 

No  scratch  marks  _3 

Total  2G 

Table  2:  Number  of  students  with  standard  and  nonstandard  orders 


For  the  purposes  of  comparing  control  regimes,  it  is  helpful  to  divide  the  nonstandard  orders 
nto  two  classes:  locally  nonstandard  orders  and  globally  nonstandard  orders.  To  do  so  requires 
that  some  notation  be  introduced. 

A  rule-based  notation  is  used  for  core  procedures  Table  3  shows  a  core  procedure.  Each 
subscripted  symbol  is  a  goal,  and  each  rule  is  a  method  for  achieving  the  goal  on  its  left  side  The 
subscripts  indicate  the  way  a  goal  passes  arguments  to  its  subgoals  in  the  case  of  subtraction, 
the  arguments  always  happen  to  be  columns  For  instance  the  rule  C,  ->A  F_,  ■  means  that  the 
column  processing  goal.  C.  can  be  achieved  for  column  i  by  achieving  three  suogcals  adding  ten 
to  the  top  digit  of  column  i  A  performing  the  borrow-trom  subgoal  on  the  next  column  ;F  and 
taking  the  difference  between  the  two  digits  of  column  i  and  writing  it  m  the  answer  position  of 
column  i  - ,  Alongside  each  rule  s  a  condition  indicating  when  the  rule  is  applicable  Thus,  the 
'uie  ,ust  stated  s  applicable  when  the  too  digit  m  column  i  is  less  than  the  bottom  digit  :T <3  ) 


'99  students  ,sec  -o  scratcn  ""a^s  Cansea^e'-t'y  <v9  can  re  i  almost  •'Otr-rj  aoout  ”-e  erder  trey  used 
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Sub  — *  Cj 

Cj  -» 

Cj  -*  A,  Fl+1  -| 

F,  -4  SD, 

F,  -+  F,.,  A,  SD, 

SD,  -  S,  D, 

Table  3: 


For  each  column! 

When  T,  2  B, 

When  T,  <  Bj 
When  T(  *  0 
When  T,  =  0 
Always 

A  core  procedure  written  in  a  rule-based  notation 


This  notation  is  designed  to  subordinate  information  which  is  not  needed  in  the  arguments 
presented  below.  However,  it  leaves  intact  the  feature  of  procedures  that  is  essential  for  the 
arguments,  which  is  that  they  are  hierarchies  of  goals.  In  fact,  this  procedure  is  recursive,  because 
the  borrow-from  goal  calls  itself  Such  hierarchical,  recursive  knowledge  representations  have 
been  widely  used  in  cognitive  science  for  representing  procedural  knowledge  (e.g.,  Newell  & 
Simon,  1972.  Anderson,  1983;  Laird,  Rosenbioom,  &  Newell,  1986).  VanLehn  (1983)  argues  from 
bug  data  to  show  that  students  in  subtraction  are  best  represented  as  having  knowledge  structures 
like  the  one  in  table  3. 

The  set  of  core  procedures  contains  30  procedures,  including  the  one  of  table  3  (VanLehn, 
1983b).  Some  core  procedures  are  complete  and  correct,  but  different  from  the  procedure  of  table 
3  m  that  they  have  slightly  different  conventions  for  borrowing  (e  g.,  for  borrowing  across  zero,  F 
calls  F^(  and  9,,  where  9,  is  a  primitive  goal  that  changes  the  top  digit  of  column  i  to  9).  Other 
procedures  are  partial  versions  of  a  taught  procedure  (e  g.,  if  borrowing  from  zero  has  not  been 
learned,  then  the  rule  F,  — >  F^,  A  SD,  is  missing).  Other  procedures  have  overgeneralized  or 
overspecialized  test  conditions  ;e  g.,  C,  borrows  when  the  top  digit  m  column  i  is  less  than  or  equal 
:o the  bottom  digit). 

A  trace  tree  is  a  way  of  displaying  the  history  of  execution  of  a  procedure  Us  'eaves  are  the 
primitive,  observable  actions  of  the  procedure  The  nonleaf  nodes  are  instances  of  nonprimitive 
goals  A  node  with  its  descendents  corresponds  to  a  rule:  the  node  is  the  left  side  of  the  rule, 
nstantiated,  and  the  descendents  correspond  to  the  subgoals  on  the  ngnt  s.de  of  the  rule 
Descendents  of  a  node  are  arranged  chronologically  so  that  the  leftmost  descendent  corresponds 
*o  the  subgoal  that  was  first  executed,  and  the  rightmost  descendent  to  the  last  subgoal.  This 
implies  that  the  leaves  are  ordered  so  that  the  primitive  actions  they  represented  occur  m  the 
■eft -to-nght  order  of  the  leaves  Figure  i  shows  a  trace  tree  for  a  standard  order  execution  of  the 


core  procedure  shown  in  table  3. 


When  a  procedure  is  executed  so  that  the  descendents  of  a  node  have  the  same  order  as 
the  right  side  of  the  rule,  then  one  of  the  standard  orders  is  generated.  This  is  a  consequence  of 
the  learning  model,  (VanLehn,  1983a;  VanLehn,  1983b)  which  constructs  core  procedures  by 
generalizing  the  example  execution  that  the  teacher  presents. 

With  the  notations  for  core  procedures  and  trace  trees  introduced,  the  subclassifications  of 
nonstandard  orders  can  now  be  defined. 

A  locally  nonstandard  order  produces  a  trace  tree  where  the  order  of  descendents  of  some 
node  does  not  correspond  to  the  right-side  ordering  of  the  corresponding  rule.  The  trace  tree  in 
figure  2  corresponds  to  a  nonstandard  order  execution,  because  the  ordering  of  subgoals  for 
borrowing  does  not  correspond  to  the  A,  Fi<t  order  of  the  right  side  of  the  borrowing  rule  of  the 
core  procedure  of  table  3.  Indeed,  none  of  the  30  core  procedures  use  the  order  shown  in  the 
trace  tree,  wherein  the  column  is  answer  before  borrowing  is  completed.  Thus,  the  protocol  shown 
the  figure,  which  corresponds  to  the  leaves  of  the  tree,  can  not  be  modelled  by  a  standard 
execution  of  any  core  procedure.  When  we  say  that  a  person  has  a  locally  nonstandard  order,  we 
mean  that  there  is  no  core  procedure  which  allows  a  standard  order  execution,  and  there  is  at  least 
one  core  procedure  that  allows  a  locally  nonstandard  order  execution. 

A  globally  nonstandard  order  produces  a  trace  tree  which  is  nonplanar,  in  that  the  branches 
cross  Figure  3  shows  a  globally  nonstandard  order  execution  for  the  core  procedure  of  table  3. 
Notice  that  the  borrowing  actions  of  all  columns  are  completed  first,  then  the  columns  are 
answered  m  left-to-right  order  There  is  no  core  procedure  that  allows  this  sequence  of  actions  to 
be  executed  in  a  standard  or  locally  nonstandard  order  so  we  say  that  t  ,s  a  globally  nonstandard 
order. 


The  mam  points  to  be  taken  from  tms  discussion  of  the  data  are  that  there  are  three  types  of 
orders,  standard,  locally  nonstandard  and  globally  nonstandardand.  and  that  ail  three  types  occur 
n  the  data  Moreover,  the  orders  often  occur  mixed  together  n  the  performance  of  a  single 


student 


Figure  3:  Trace  tree  tor  a  giocaiiy  nonstandard  execution 
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4.  Which  control  regimes  are  consistent  with  the  data? 

Each  of  the  four  control  regimes  will  be  considered  in  turn  for  their  ability  to  produce 
nonstandard  orders. 

The  deterministic  stack  regime.  A  deterministic  stack  regime  is  the  weakest  control  regime. 
When  a  subgoal  is  being  executed,  the  goal  resides  on  a  stack,  along  with  an  index  into  its  list  of 
subgoals.  The  index  indicates  which  subgoal  to  call  next  when  the  present  subgoal  is  finished. 
Figure  4  shows  the  stack  at  each  cycle  of  interpretation  during  a  standard  order  execution  of  the 
procedure  of  table  3  on  the  problem  306-28. 

in  a  deterministic  stack  regime,  the  order  of  execution  of  subgoals  is  fixed  by  the  contents  ot 
the  procedure.  The  order  is  the  order  in  which  the  subgoals  were  listed  on  the  right  side  of  the  rule. 
However,  this  fixed  ordering  makes  it  impossible  for  the  deterministic  stack  regime  to  model  even 
the  locally  nonstandard  orders,  let  alone  the  globally  nonstandard  ones. 

Nondeterministic  stack  regime.  The  nondetermistic  stack  regime  is  the  next  step  up  in  power, 
from  the  deterministic  stack  regime,  it  does  not  fix  the  order  of  subgoal  execution  in  advance,  but 
instead  employs  a  scheduler  that  chooses  which  subgoal  to  execute.  Thus,  when  a  rule  is  about  to 
be  executed  for  the  first  time,  the  scheduler  chooses  which  subgoal  from  the  right  side  to  execute 
first.  The  chosen  subgoal  is  instantiated,  the  current  goal  is  pushed  onto  the  stack,  and  execution 
of  the  subgoal  begins.  When  its  execution  is  finished,  scheduler  chooses  one  of  the  remaining 
subgoals  as  the  second  one  to  be  executed.  When  all  the  subgoals  have  been  executed,  the  goal 
is  popped  and  control  returns  to  the  goal  that  called  it. 

The  implementation  of  GPS  described  by  Newell  and  Simon  (1972)  obeys  a  nondeterministic 
stack  regime  and  uses  a  simple  scheduler  based  on  a  total  order  of  goal  types.  When  a  goal  is 
begun  or  continued,  the  subgoais  are  categorized  and  their  types  are  looked  up  in  a  list.  The 
subgoal  whose  type  occurs  first  m  tne  list  is  chosen  for  execution  if  two  subgoals  tie.  because  they 
have  the  same  type,  then  the  model  is  underdetermined  at  that  point,  it  does  not  predict  which 
choice  the  subiect  will  make  This  is  why  the  control  regime  is  called  a  nondeterministic  stack 
regime  The  list  of  goal  types  s  a  parameter  to  the  model  it  represents  a  scheduling  strategy. 
Since  the  types  are  task  specific,  the  scheduling  strategy  is  necessarily  task  specific  as  well. 
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Figure  4:  Three  control  regimes  executing,  A . indicates  the  active  goal 


For  subtraction,  a  total  order  ot  goal  typas  would  merely  cause  emulation  of  the  deterministic 
stack  regime,  so  a  slightly  more  powerful  scheduling  mechanism  is  used.  Goal  types  are  partially 
ordered  instead  of  totally  ordered.  A  scheduling  strategy  is  notated  by  a  set  of  transitive, 
antisymetric  ordering  constraints.  For  instance,  the  ordering  relation  S,>D,  means  that  slashing  is 
preferred  over  decrementing  given  that  both  refer  to  the  same  column,  i.  This  constraint  will  cause 
the  scheduler  to  first  pick  S2  off  the  agenda  when  both  S2  and  D2  are  on  it  unless,  ot  course,  there 
is  some  other  goal  on  the  agenda  that  is  even  more  highly  preferred  then  S2.  Once  S2  has  been 
executed  and  is  therefore  off  the  agenda,  the  scheduler  is  free  to  pick  D2.  Another  example  is  the 
constraint  CI>C,.1,  which  means  that  the  column  processing  goals  will  to  be  processed  in  right  to 
left  order. 

The  scheduler  uses  the  partial  order  established  by  such  constraints  to  choose  one  of  the 
remaining  subgoals  of  a  goal  whenever  that  goal  is  begun  or  continued.  Because  the  order  is 
partial  and  not  total,  it  is  possible  for  two  goals  to  tie  for  first  place  even  when  they  are  of  different 
types.  For  instance,  suppose  that  A  >-,  is  the  only  constraint  relevant  to  the  borrowing  goal's 
subgoals.  A,,  F  and  When  the  goal  is  begun,  two  goals  will  tie  for  first  place,  the  add-ten 
subgoal.  A,,  and  the  borrow-from  subgoal,  F^,.  Thus,  the  model  does  not  predict  which  goal  will 
be  choosen  by  the  subject.  It  will  be  consistent  with  either  one. 

This  is.  of  course,  not  the  only  way  to  notate  scheduling  strategies.  It  is  merely  the  next  step 
up  in  power  from  GPS's  notation.  Indeed,  it  will  turn  out  later  that  this  notation  is  not  powerful 
enough  to  represent  some  of  the  regularities  in  the  data.  However,  the  notation  does  allow  a 
precise  comparision  of  control  regimes,  which  is  all  !f  jt  we  need  of  it  for  the  purposes  of  this 
paper 

The  nondetermimstic  stack  regime  can  be  parametenzed  so  that  it  will  generate  the  locally 
nonstandard  orders  as  well  as  the  standard  ones  For  instance,  in  order  to  generate  the 
nonstandard  order  illustrated  in  figure  2.  the  constraints  ->F_.  and  a  >-  are  used  to  order  the 
subgoals  of  borrowing.  However,  there  is  no  way  to  parameterize  the  mccei  so  that  the  globally 
nonstandard  orders  are  generated 

The  agenda  regime  Another  type  of  control  regime  uses  an  agenda  as  its  goal  memory  An 
agenda  is  an  unordered  set  of  instantiated  goals  When  a  goal  is  executed,  it  is  f.rst  removed  from 
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the  agenda,  then  all  its  subgoals  are  instantiated  and  placed  on  the  agenda.  Thus,  an  agenda 
represents  a  set  of  pending  goals,  i.e.,  an  unordered  plan  for  all  the  things  that  need  to  be  done  to 
finish  the  problem.  Figure  4  shows  the  agenda  during  execution  of  the  procedure  of  figure  3. 

An  agenda  1  •'gime  requires  a  scheduler  to  choose  which  of  the  goals  on  the  agenda  to 
execute  next.  The  agenda  regimes  in  Al  have  tended  to  use  very  complicated  schedulers  (Nii, 
1986).  Since  this  scheduler  is  part  of  a  comparision  between  control  regimes,  it  makes  sense  to 
use  the  same  scheduler  as  was  used  in  the  nondeterministic  stack  regime.  This  means  that 
differences  in  generative  power  will  be  due  to  the  control  regime  and  not  differences  in  the  power  of 
the  scheduler. 

An  agenda  regime  allows  enough  flexibility  to  model  both  globally  and  locally  nonstandard 
orders.  The  key  to  this  flexibility  is  that  the  agenda  regime  can  mix  the  execution  of  subgoals  from 
two  different  goals,  because  the  agenda  allows  access  to  all  pending  subgoals.  The 
nondeterministic  stack  regime  allows  access  to  only  the  most  recent  goal's  subgoals.  As  an 
illustration  of  this  power,  figure  5  shows  the  agenda  at  each  step  of  a  globally  nonstandard 
execution  order.  The  problem  345-189  is  solved  in  two  passes,  with  all  the  borrowing  on  the  first 
pass  and  all  the  column  differences  on  the  second  pass.  Table  4  gives  a  scheduling  strategy  that 
will  produce  this  order  The  agenda  regime  is  the  first  of  the  regimes  to  be  discussed  that  has 
sufficient  power  to  model  all  the  observed  execution  orders. 

The  tree  regime.  The  fourth  type  w  control  regime  for  recursive  programs  is  based  on 
keeping  a  tree  of  instantiated  goals.  When  the  procedure  has  finished,  the  tree  is  exactly  the  trace 
tree  for  the  procedure  on  that  problem.  During  execution,  this  tree  is  constructed  incrementally.  A 
system  of  markers  is  used  to  differentiate  pending  goals  from  those  that  have  been  executed.  As 
with  the  agenda  regime,  a  scheduler  picks  which  of  the  pending  goals  to  execute.  Again,  we  will 
assume  that  the  scheduler  is  guided  by  a  scheduling  strategy  that  is  represented  as  a  partial  order. 
Figure  4  shows  a  few  cycles  of  interpretation  using  a  tree  regime. 

The  tree  regime  can  emulate  the  agenda  regime  because  whenever  the  agenda  always 
corresponds  to  the  set  of  goals  m  the  tree  that  are  marked  pending.  In  particular,  both  the  tree 
regime  and  the  agenda  regime  can  handle  all  three  types  of  orders.  The  agenda  regime  is  slightly 
~cre  elegant  because  it  maintains  the  minimal  temporary  state  needed  to  do  its  job.  The  extra 
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Figure  5:  The  fwo-cass  strategy  running  on  an  agenda  regime 


C,>C^,  Do  columns  right-to-lett 

Do  column  differences  left-to-nght 

S;>D,  Do  slash  before  decrementing 

X  ^,>A|  for  X=F,S,D,A  Do  all  borrowing-from  before  add  10 

X,>— ,,  for  X=B,A,F.SD,S,D.Z,  or  9  Do  column  differences  last 

Table  4:  Constraints  for  the  two-pass  scheduling  strategy 


nodes  in  the  tree  serve  no  purpose  as  far  as  scheduling  is  concerned.  However,  this  is  not  a 
sufficient  reason  to  prefer  the  agenda  regime  over  the  tree  regime.  More  empirical  evidence  is 
needed.  Fortunately,  such  evidence  is  already  at  hand. 

In  our  earlier  work  on  repair  (Brown  &  VanLehn,  1980;  VanLehn,  1983c),  it  was  shown  that  a 
common  repair  is  to  retreat  to  a  super-goal  of  the  goal  where  the  impasse  occurs  and  resume 
interpretation  there  This  kind  of  repair  is  called  hierarchical  backup,  it  is  different  from 
chronological  backup  (the  kind  used  in  (Newell  &  Simon,  1972))  or  dependency  directed  backup 
;deKleer,  1986).  It  resembles  a  nonlocal  “return''  in  hierarchical  programming  languages  (e.g.,  the 
catch-throw  construct  in  CommonL/sp).  In  order  to  function  properly,  hierarchical  backup  requires 
that  the  architecture  maintain  either  a  stack  or  tree  of  goals  so  that  hierarchical  backup  can  easily 
access  a  supergoal  of  the  currently  active  goal.  With  an  agenda,  which  has  only  pending  goals, 
backing  up  to  a  supergoal  would  require  a  reverse  interpretation  process  to  reconstruct  the  chain  of 
supergoals  that  called  the  currently  active  goal.  Reverse  interpretation  is  complex  because  it  can 
be  non-determimstic.  if  a  goal  is  called  from  two  different  supergoals,  the  reverse  interpreter  must 
guess  which  one  was  in  fact  the  one  that  called  the  goal.  In  short,  the  agenda  just  does  not  have 
as  much  information  as  the  tree,  and  moreover,  that  missing  information  is  sometimes  crucial  for 
the  backup  repair  The  backup  evidence  shows  that  the  architecture  obeys  a  tree  regime  rather 
•nan  an  agenda  regime. 

it  seems  that  the  tree-based  control  regime  is  uniquely  capable  of  modelling  both  the 


ordering  data  and  the  repair  data  We  can  tentatively  conclude  that  the  tree-based  control  regime 
,s  obeyed  by  the  mental  architecture  that  human  procedures  are  executed  by. 
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5.  How  well  does  the  tree  regime  fit  the  protocol  data? 

So  far,  the  argument  has  been  based  on  a  qualitative  treatment  of  the  protocol  data,  it  was 
asserted  that  some  subjects,  on  some  problems,  used  execution  orders  that  only  the  tree  and 
agenda  regimes  can  model.  This  section  fits  the  tree  regime  to  the  protocol  data  in  order  to 
determine  how  much  of  the  variability  in  student's  execution  orders  can  be  captured.  This  will  yield 
an  intuitive  assessment  of  the  absolute  quality  of  the  tree  regime,  rather  than  its  quality  relative  to 
•the  other  control  regimes.  It  will  also  demonstrate  the  instability  of  subject’s  scheduling  strategies 
over  time. 

5.1.  The  fitting  procedure 

This  section  describes  how  the  tree  regime  is  fit  to  the  data.  There  are  three  parameters  that 
must  be  given  values:  (1)  the  procedure,  (2)  the  scheduling  strategy,  and  (3)  the  repairs  to  any 
impasses  that  occur.  The  parameterization  method  is  to  first  choose  a  procedure  and  repairs,  then 
fit  a  scheduling  strategy.  In  all  cases,  the  choice  of  procedure  and  repairs  was  quite  clear  cut,  so  it 
was  done  by  hand.  Table  5  shows  the  procedures  assigned  to  each  student. 

The  choice  of  scheduling  strategy  was  less  clear.  We  had  to  find  a  good  fit  by  running  the 
model  with  our  best  guess  at  a  set  of  constraints,  then  examining  the  residuals.  Residuals  are 
places  where  the  model  and  the  protocol  disagree.  There  are  two  kinds  of  residuals: 

•  Underprediction:  More  than  one  pending  goal  is  maximal  according  to  the  constraint 
set,  and  the  student's  choice  is  among  the  maximal  items.  The  model  partially 
explains  the  student's  choice,  but  falls  short  of  exactly  predicting  it. 

•  Misprediction:  The  student's  choice  is  not  among  the  maximal  pending  goals.  The 
model  mispredicts  the  student's  behavior. 

To  arrive  at  a  numerical  evaluation  of  the  model's  fit  to  a  student's  protocol,  we  counted  the  cases 
of  underprediction  and  misprediction,  in  general,  these  two  counts  are  inversely  related,  if  the 
underprediction  count  is  high,  then  adding  ordering  constraints  will  bring  that  count  down. 
However,  this  usually  causes  the  mode!  to  make  some  wrong  choices,  driving  the  misprediction 
count  up 

in  general,  we  preferred  to  minimize  mispredictions.  A  model  that  predicts  that  the  students 
sometimes  act  on  guesses  is  preferable  to  a  model  that  predicts  that  they  have  a  scheduling 
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C!  'l 
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F  -*  SDj 

P,  -»  P„,  A,  SO, 

Janine  and  Tanya 
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Hilda 

ci  'i 

C,  -»  F 

p,  -s 

F,  — »  F 


P,*i  A, 


Paul  and  Pete 
C,  -* 


7T* 

>  ‘ 

c,  -  Ai  F,»i  -i 

C,  -  Ft.,  A,  - 

■»  > 

F,  -»  SD, 

F,  “»  Sj  D, 

is 
• « • 

F,  -  9,  F,.i 

P,  ^  F,*t  A,  S,  D, 

& 

Robby 

Primitives 

■ 

c, -»•, 

-,  Take  difference  in  column 

c,  ->  Ai  F,*t 

A,  Add  ten  to  top  of  column  i 

Sj  Slash  top  digit  of  column  i 
D,  Decrement  top  digit  of  column  i 
SDj  Slash  and  decrement  column  i 
9,  Change  top  digit  of  column  i  to  9 


A,  F,., 


Table  5:  Procedures  for  the  Nonstandard  order  students. 

Hilda,  Trina  and  Robby  have  bugs.  The  others  students  are  bug-free. 


strategy  that  they  do  not  obey.8 

Table  6  shows  the  best  fitting  models  for  the  eight  nonstandard  order  students.  The 
scheduling  strategies  for  all  the  students  except  Tanya  included  ordering  relations  needed  for 
standard  procedures  to  be  executed  in  such  a  way  that  they  produce  a  correct  answer.  We  call 
these  the  base  set  of  relations.  Table  7  lists  them.  They  are  included  by  reference  in  the  figure  6. 
Tanya's  constraint  set  included  some,  but  not  all.  of  the  base  relations. 

Table  6  shows  that  the  number  of  underdetermined  choices  is  high,  it  ranges  from  8°0  to 
5i3o  of  the  total  number  of  choices  made  during  the  protocol,  with  a  mean  of  33%.  This  is  not 
good.  Taken  literally,  it  means  that  the  students  are  guessing  one-third  of  the  time 
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SDi.,>c,.  9i>F,*i.  9, base 

Paul 

0% 

19  6% 

158 

Ai*i>Ai.  F,.i>Ai.  S,,1>Ai, 

A,>C,.i.  -,>C^.  base 

Pete 

0% 

51  0% 

100 

Fi*i>Ar  Si»i >Ai'  base 

Robby 

0% 

43.4% 

145 

SDi.i>Ap  Ai.,>-,.  F, 

SDh>-,.  base 

Tanya 

0% 

7  9% 

114 

SDm>A„9,>F..,, 

X,> lor  XsF,  SD,  A.  S,  or  9, 
for  X=F,  SD,  A,  S,  or  9. 


Trma  1.4%  44. 7%  143  A,  >  — (.  SD,_f>  —  SD  >A,,  base 


Table  6:  Best  fitting  agenda  models  for  the  eight  nonstandard  order  students. 
Miss  »  Percentage  of  total  choices  mispredicted  by  the  model. 
Under  =  Percentage  of  total  choice  underpredicted  by  the  model. 
Total  =  Number  of  agenda  choices  total. 


C>C., 

Subtract  columns  from  right  to  left 

X,>C,  for  X=F  s.  D.  SO.  9  A 

Change  column  before  testing  T>B 

A  >  - 

Avoid  subtracting  larger  from  smaller 

S  >D, 

Slash  before  decrement 

A  >SD, 

Avoid  decrementing  0  during  BFZ 

A  >S. 

AddiO  before  slash  during  BFZ 

A  >D. 

AddiO  before  decrement  during  BFZ 

Table  7: 

The  base  set  of  relations 

When  we  examined  the  students  supposed  guesses,  we  discovered  an  underlying  pattern 
Some  students  use  more  than  one  execution  strategy  To  illustrate  it.  well  consider  one  student. 
paul.  m  detail.  A  facsimile  of  Paul's  test  sheet  is  included  as  figure  5  His  protocols  are  given  m 
table  3. 
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Before  going  over  Paul's  protocol  in  detail,  it  should  be  mentioned  that  the  protocol  data  tor 
all  the  students  has  been  edited  in  order  to  remove  actions  that  can  not  be  accounted  for  by  the 
simple  core  procedures  we  used.  For  instance,  m  problem  i,  Paul  actually  did  -2  -2  -3  He 

rewrote  his  answers  to  the  units  and  the  tens  columns,  probably  because  he  thought  that  they  were 
illegible  (they  looked  illegible  to  us,  too.).  The  simple  core  procedures  produced  by  the  learning 
model  can  not  represent  these  extra  actions,  so  they  were  removed  from  the  data.  The  appendix 
presents  the  raw  data  alongside  the  edited  data,  and  explains  why  each  edit  was  made.  We  feel 
that  the  cleaned  up  data  remains  adequate  for  testing  the  fit  of  the  tree  regime. 
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Figure  6:  Paul's  test 


On  some  problems  (problems  12  and  most  of  problem  11).  Paul  consistently  decrements 
before  adding  ten  dunng  borrowing.  On  other  problems  (problems  4.  5.  5.  3  and  9).  he  adds  ten 
and  subtracts  the  column  before  decrementing  On  problems  7  and  1 0.  ne  sometimes  decrements 
first  and  sometimes  adds  ten  first  The  remaining  problems  iprooiems  ■  through  3;  do  not  require 
any  borrowing,  so  we  can  not  tell  what  scheduling  strategy  he  was  usmg  for  mem  Consequently, 
the  largest  constraint  set  that  avoids  mispredictions  is  one  that  dees  not  take  a  stand  on  now  the 
decrement  operation  is  ordered  with  respect  to  the  other  borrowing  operations  This  five-memoer 
constraint  set  is  the  one  shown  in  in  table  5  it  exactly  predicts  Paul  s  performance  only  on  the 


'IvlvV  •/- 


IS 


was? 


12  3  4 


4.  S2  A^  D2  ”2 

5.  S2  A,  D2  -2 

6.  S2  A,  D2  -2  -3 

7.  S2  A,  D2  S3  A2  D3  -2  S4  A3  -3  D4  -4 

8.  S2  A,  D2  S3  A2  -2  D3  -3 

9.  S2  A,  02  S3  A2  -2  D3  -3  -4 
10  S2  A,  D2  -2  S4  A3  D4  -3  -4 

1 1 .  S2  A,  D2  s5  d5  A4  S4  d4  a3  S3  d3  a2  -2  -3  -4  -5 

1 2.  S4  D4  A3  S3  D3  A2  S2  D2A,  -1  - 2  '3  "4 

Table  8:  Paul’s  protocols 


three  problems  that  do  not  require  borrowing.  However,  Paul’s  choices  on  the  test  can  be  almost 


exactly  predicted  if  the  model  employs  one  constraint  set  (i.e. ,  the  original  five  plus  on  - 


some  problems  (problems  4, 5, 6, 8  and  9,  and  parts  of  problems  7, 10  and  11),  a  different  constraint 


set  (i.e.,  the  original  five  plus  Ou1>A,)  on  others  (problem  12  and  the  remainder  of  problem  11) 


and  a  third  constraint  set  on  two  columns  in  the  middle  of  problems  7  and  10. 


Paul  is  not  alternating  randomly  among  the  possible  legal  orderings  of  borrowing  operations, 
as  the  scheduling  strategy  of  figure  6  predicts.  There  are  several  more  permutations  of  borrowing 
operations  than  the  three  that  Paul  uses.9  Paul  has  definite  preferences  about  which  orders  to  do 


borrowing,  and  these  constraints  sets  capture  them  exactly 


We  found  similar  patterns  for  six  of  the  eight  students  with  nonstandard  orders  Table  9 


snows  the  results  of  fitting  the  model  to  minimize  underprediction  by  using  multiple  constraint  sets 


The  appendix  presents  the  strategies  fit  to  each  student. 


in  three  cases  (Hilda.  Paul  and  Tanyat,  we  found  sets  of  constraints  that  would  yield  an 


exact  match,  indicating  that  the  students  were  alternating  among  multiple  scheduling  strategies,  in 


5~"er9  are  h*ree  C'Oers  that  ®aui  aces  10S  use  r )  acd  ten  s  ash  cecrement.  aifference  (2;  aod  ten  s:ash.  afte'ence 
-jecre'T'e^t  3>acja:en  <2;'*er orce  sas~i  oec'e^e^t 


•  m.  "*V  *  k  *  •.  «  »  "  - 


Subject 

Sets 

Miss 

Under 

& 

Angela 

i 

3.6% 

1  4% 

Hilda 

3 

0% 

0% 

Janme 

4 

0% 

7  1  % 

£ 

Paul 

3 

0% 

0% 

Pete 

3 

0% 

5% 

w 

Robby 

3 

0% 

5  2% 

Tanya 

2 

0% 

0% 

_ 

Tnna 

1 

8  4% 

0% 

h 


TaWe  9:  Fits,  allowing  multiple  scheduling  strategies 
Sets  -  numper  ot  constraint  sets. 

Miss  =  Percentage  of  agenda  choices  mispredicted. 
Under  -  Percentage  of  agenda  choices  underpredicted. 
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three  other  cases  (Janme,  Pete  and  floppy),  the  use  of  multiple  strategies  instead  of  one  helped 
the  fit.  but  did  not  yield  an  exact  match,  in  the  remaining  two  cases  (Angela  and  Trina),  using  two 
or  more  strategies  did  not  help  the  fit  much  at  all. 


Fit  with  one  scheduling  strategy 

Fit  with  two  or  three  scheduling  strategies 

Poor  fit 

Total 

Table  10:  Summary  of  the  fit  of  the  agenda  model 


The  overall  fit  of  the  model  is  summarized  in  table  10.  it  shows  two  basic  findings.  First, 
good  fits  were  obtained  for  all  the  students  except  Trina.  We  judge  that  the  model's  fit  to  Trina  is  a 
'poor"  fit,  although  it  is  not  as  bad  as  the  fit  of,  say,  the  deterministic  stack  regime  to  her  protocol 
.Ve  just  can  not  see  any  pattern  m  Trina  s  performance.  The  second  Dasic  finding  is  that  75%  of 
•he  nonstandard  order  students  seem  to  be  using  multiple  scheduling  strategies 

We  conclude  that  the  tree  regime  allows  an  excellent  fit  to  the  data,  but  that  it  fits  .veil  only  at 
the  cost  of  adding  a  mystery:  what  causes  students  to  shift  strategies7  The  next  section  presents 
our  speculations  on  this  issue. 


6.  Multiple  strategies:  what,  when  and  why? 

At  tnis  time,  we  do  not  have  an  explanation  ot  why  some  students  have  multiple  strategies 
and  what  causes  them  to  shift  among  them.  However,  there  are  some  interesting  aspects  to  the 
multiple-strategy  data  that  hint  at  the  underlying  causes. 

The  first  question  to  ask  is  what  sort  of  strategies  tend  to  appear  together  In  one  student's 
behavior.  There  is  no  way  to  answer  such  a  clustering  question  in  a  principled  manner,  so  a 
heuristic,  intuitive  classification  will  have  to  suffice.  The  eight  subjects  can  be  classified  into  two 
groups  of  four  each.  The  first  group  (Hilda,  Paul,  Robby,  and  Tanya)  have  strategies  that  differ 
only  in  the  way  they  order  the  subgoals  of  borrowing,  whereas  the  second  group  (Angela,  Pete, 
Janine  and  Trina)  have  strategies  that  differ  in  the  way  they  order  the  column  subgoals.  For 
example.  Tanya  is  m  the  first,  "borrow  variations"'  group  because  all  her  strategies  are  similar  She 
always  does  all  her  borrowing  before  any  column  is  answered.  She  borrows  from  right  to  left,  then 
she  answers  columns  from  right  to  left.  However,  she  uses  two  strategies  for  borrowing. 
Sometimes  she  adds  ten  then  borrows-from.  and  other  times  she  borrows-from  then  adds  ten.  So 
her  strategies  are  minor  variations  of  each  other. 

We  conjecture  that  students  in  the  "'borrow  variations"  group  actually  have  a  single,  uniform 
strategy,  but  our  representation  for  strategies  can  not  express  that  strategy  in  fact,  we  found 
uniform  strategies  for  two  of  the  students.  Paul  and  Tanya.  Tanya  s  strategy  can  be  expressed  as 
if  1=1  then  A  >F  ,  else  if  i=2  then  F  >A  .  Paul's  strategy  (except  for  problems  7  and  10)  seems  to  be 
if  borrowing  from  zero,  then  D_,>A.  else  \>D_,  These  strategies  are  conditional  on  the  state  ot 
the  problem  solving,  so  the  partial  order  cannot  express  them 

We  have  developed  a  more  expressive  representation  for  scheduling  strategies  as  well  as  a 
program  that  will  automatically  fit  a  strategy,  expressed  n  this  "epresemat'cn  to  me  suciect  s 
oehavior  tVanLehn  S  Gariick  *987  With  this  tool  we  hope  to  discover  creese  .rn-form  srateg'es 
‘or  all  the  students  m  me  borrow  vanat-ens  group,  which  s  a  *-r$:  step  ‘c.va'os  e>c  a  mng  wrat 
students  are  do  ~g  a-d  w-v 

-or  ^stance  "a-mg  ~c'ed  "~v  3  au's  sfaregy  s  ccrrcw  rq  '-cm  zero  "---  Z  . ,>a  ~  se 
-  >C  ..  "  re  .:3n  mmed'arey  see  a  sr-pe  sng  n;-v.-e  eoanat'cn  3aui  does  a  sta-cam  order 
on  bcrrcw-‘'om-z9ro  orooie"'s  because  “e  "I'S  these  prop  e^s  are  -a-q  a-d  ~e  o^  d  -o'ease 
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ms  chances  of  getting  them  right  by  using  the  teacher’s  strategy.  This  implies  that  he  knows  two 
distinct  strategies,  his  and  the  teacher  s,  and  uses  meta-cognitive  reasoning  to  select  one.  This 
explanation  is  consistent  with  a  phenomenon  that  has  been  occasionally  observed  but  has  not  yet 
received  systematic  investigation.  Some  students  seem  to  have  two  or  more  distinct  procedures. 
For  instance,  one  student  we  interviewed  answered  a  whole  test  by  taking  the  absolute  difference 
n  every  column,  even  those  requiring  borrowing.  Afterwards,  the  interviewer  asked  her  if  she  knew 
about  borrowing.  She  said  she  did.  and  showed  us  by  answering  two  borrowing  problems 
correctly  Resnick  and  Gmanson  ;i986i  observed  several  sublets  who  seem  to  have  multiple 
crccedures  Cnly  a  little  prompting  ;their  "prohibition"  condition)  sufficed  to  make  them  switch  from 
a  buggy  procedure  to  a  correct  one.  Paul  is  similar  to  these  students  in  that  he  seems  to  know  two 
c  st  net  strategies  believes  they  both  give  correct  answers,  but  have  different  resource  or  accuracy 
characteristics 

if  this  coniectured  multiplicity  of  procedures  and  strategies  withstands  empirical  testing,  then 
one  of  the  central  assumptions  of  cognitive  modelling  must  be  modified.  It  has  always  been 
assumed  that  sublets  in  skill  acquisition  and  problem  solving  experiments  have  just  one 
'-cwiedge  structure,  but  these  findings  indicate  there  might  be  several,  with  a  "big  switch"  that 
Se  ects  one  or  the  other 

A'thougn  the  patterns  of  behavor  in  the  borrow  variations  group  may  soon  yield  to 
emanations  with  important  theoretical  implications,  the  patterns  of  behavior  in  the  other  group 
A'-qe  a  uanire  3ete  and  Tnnai  are  more  difficult  to  understand  intuitively,  it  appears  that  all  four 
vmems  start  out  wth  a  standard  order  strategy  at  the  beginning  of  the  test,  then  become 
"'easing  '■orstandard  towards  the  end  Ail  four  ended  with  a  strategy  that  involves  answering 
ccumne  n  a  "wild"  order,  e  one  that  is  neither  nght-to-teft  nor  left-to-right.  One  possible 
i'Z  3” at  on  ‘or  this  behavior  s  that  they  become  increasingly  confident  as  the  test  progresses  and 
c~gr1  '0  shew  off  their  skills  '"ere  are  of  course  other  equally  plausible  explanations 
3 :  "s ■cer3Cie  empirical  and  mecret-cai  work  wni  needed  in  order  to  understand  and  differentiate 
' exp(an.at  prs 


6.1.  Execution  of  procedures  as  search 

This  section  is  speculative,  it  introduces  a  generalization  of  the  tree-based  control  regime 
that  makes  intuitive  sense  and  connects  the  results  presented  earlier  with  the  existing  litterature  on 
puzzle  solving.  However,  the  results  do  not  conclusively  support  this  model  over  a  simpler  tree- 
based  model.  This  conjecture  should  be  understood  as  an  outline  for  future  research. 

Under  the  tree  regime,  two  different  mechanisms  are  searching  the  goal  tree  -  one  is  the 
scheduler,  and  the  other  is  the  mechanism  that  does  repairs.  Both  search  for  a  goal  to  perform 
next  We  coniecture  that  they  are  the  same  cognitive  process  because  the  student  is  trying  to 
solve  the  same  problem.  The  student's  problem  is  "which  goal  should  I  do  next?"  and  it  should  be 
solved  in  such  a  way  that  the  resulting  solutions  should  look  on  paper  like  the  standard  procedure 
had  been  executed  m  a  standard  order.  As  both  scheduling  and  repair  seem  to  respect  this 
constraint,  it  seems  likely  that  they  result  from  the  same  process,  rather  than  being  two  different 
processes,  as  they  are  under  the  tree  regime. 

The  proposed  process  is  like  classical  problem  solving,  except  that  the  problem  to  be  solved 
is  not  at  the  level  of  the  task,  but  is  meta  to  the  task.  In  particular,  the  problem  is  not  to  solve  a 
subtraction  problem  but  to  find  a  goal  in  the  subtraction  procedure  to  execute. 

As  a  species  of  imeta-)  problem  solving,  one  would  expect  the  cognitive  process  to  have 
some  of  the  attributes  found  in  ordinary,  base-level  problem  solving.  In  particular,  just  as  some 
subiects  alternate  among  search  strategies  while  solving  puzzles  (Newell  &  Simon,  1972),  one 
would  expect  to  find  subjects  switching  among  search  strategies  while  doing  (meta-level  problem 
solving  As  mentioned  earlier,  most  of  the  nonstandard  order  subjects  do  shift  among  strategies. 
This  does  not  explain  why  they  switch  strategies,  nor  where  they  got  the  strategies  that  they  switch 
among.  However,  it  is  somehow  comforting  that  the  same  familiar  mysteries  appear  in  both  meta- 
eve!  and  base-level  problem  solving. 

This  view  of  procedure  execution  seems  consistent  with  observations  by  Sucrman  and 
Wynn  1984,  n  their  study  of  office  procedures  ’  They  studied  clerks  m  a  customer  service  office 
"■‘'ey  fcund  that  much  of  the  daily  work  of  the  clerks  was  not  simp'y  following  the  prescribed  office 
crccedures  i  e  the  ones  found  m  the  procedure  manuals  of  the  corporation;  Although  some  tasks 
were  accomplished  accorcmg  to  standard  procedure,  much  of  the  time  went  into  handling  cases 


where  the  prescribed  procedure  could  not  be  followed  exactly.  In  such  cases,  the  clerks  may  do 
some  complicated  problem  solving  so  that  it  will  appear  that  the  prescribed  procedure  has  been 
followed.  Suchman  and  Wynn  recorded  the  following  account  of  a  clerk  who  attempted  to  get  a 
customer  to  pay  a  bill  when  the  bill  is  incorrect: 


Okay,  you  call  the  buyer,  the  buyer  says,  um.  the  reason  why  I'm  not  paying  this  is,  I  said  I  would 
oay  twenty  dollars  and  seventy -three  cents  <or  a  carton,  not  twenty-four  dollars  and  seventy-two 
cents,  wnich  you  Dill  me  on  this  five  thousand  dollar  shipment  of  caper.  So  then  you  say,  that’s  all  I 
need  to  know,  et  me  get  back  with  you.  You  get  back,  you  go  through  your  billing  system,  you  try  to 
*>nd  out.  you  xnow.  now  it  (pause):  In  the  meantime,  let's  suppose  time  is  running  out  and  you  do 
•'ot  nave  t  me  to  get  a  oiilmg  adjustment  through  So  you  got  to  sit  there  and  think,  How  can  I  get 
this  oerson  to  oay  this  invoice7  It's  wrong,  they  got  the  wrong  PO,  they  billed  them  wrong,  accounts 
oayaoie  coesn  t  want  to  oo  anything  witn  ,t.  So  you  call  them  Dack  ud  ana  say,  I'm  not  ashing  you  to 
oay  something  mat  s  net  oue  What  I  want  you  to  do  is  pay  ipause)  according  to  your  PO.  Pay  the 
nvoice  snort,  CKay1  T~en  -e  says,  I  wul  not  pay  that  invoice  short  because  I've  had  too  many 
problems  w  to  to  at.  Un  ess  I  get  a  typed  nvo.ee  from  you  specifically.  So  you  sit  there  and  think,  I 
cant  go  t"'Ough  me  oiilmg  system,  ts  too  ate.  I  can  tyoe  them  an  invoice.  Set  the  system  going 
rn'ocgn  me  oi  mg  system  at  me  same  time  Coordinate  that  so  when  he  pays  the  checK  short,  there 
w  oe  a  oaiance  on  me  account  Wren  the  credit  issues  through  I'll  have  the  billing  department 
"aid  mat  cred  t.  cenver  that  credit  to  me,  not  aenver  it  to  the  customer  cause  the  customer  will 
wcncer  wny  am  I  getting  toe  credit  f  they  think  they're  already  gonna  receive  a  bill,  right?  Then  I 
wo„;d  ,ust  c  ean  up  their  account  ater.  But  n  the  meantime. ...(Suchman  &  Wynn,  1984,  pg.  34) 

This  episode  dramatically  illustrates  how  complicated  the  problem  solving  of  procedure 
following  can  be.  This  clerk  is  clearly  an  expert  at  it.  Our  current  conjecture  is  that  nonstandard 
orders  and  repair,  which  we  have  observed  m  our  studies  of  subtraction,  are  just  simple  forms  of 
this  type  of  problem  solving. 
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Appendix: 


This  appendix  presents  the  protocols  of  each  of  the  eight  subjects  exhibiting  a 
nonstandard  order,  and  our  analyses  of  them.  There  are  eight  sections,  one  for  each 
subject.  Each  section  has  three  subsections:  1)  the  subject's  protocol,  annotated  to 
indicate  our  idealizations  of  it;  2)  the  constraint  sets  for  each  scheduling  strategy;  and 
3)  a  figure  showing  how  each  strategy  fares  on  predicting  the  subject's  agenda  choices, 
i  he  figures  require  some  explanation.  The  large  tick  marks  indicate  agenda  selections. 
The  vertical  stripe  beneath  a  small  tick  mark  is  black  if  the  strategy  correctly  predicts 
the  subject's  agenda  selection,  and  white  if  it  dees  not.  In  cases  where  the  subject  seems 
to  use  multiple  strategies,  the  figure  also  shows  a  bar.  labelled  "Union,-  that  shows  the 
best  fit  one  can  obtain  by  assuming  ;he  subject  switches  strategies. 


SD  2  » A i , - 1  , -  2  j 


SD  2»  A  1 ,  *  f ,  *  2 1 


SD  2,  A1 ,  -1  , 


[SD  3,  A2  ,  SD2 ,  A1  ,  ,  <write  1  in  column  2>  ], 

'  2  >  ‘ 3  .  *4 

•  We  count  it  a  slip  that  she  borrows  starting  in  column  1 
wrth  5  over  3;  so  the  idealized  protocol  does  *  i .  *2  •  *3  .  *4  ■ 


SD  35  A  21  *  2  ’  *  1  * 


SD  2,  A  1 ,  - 1  1  [■  2  1»  *  3 

•  She  does  a  -2  instead  ot  initiating  a  borrow;  the  idealized 
protocol  does  the  borrow. 


SD  2,  A  1 ,  -  1  ,  SD  3,  A  2,  *  2>  ^  4'  ^  3i  ‘  3  1  ■  4 


SD  2>  A  1 ,  •  1  ,  [-  2  ]»  SD  4.  A  3 ,  -  3  ,  -4 

•  We  count  it  a  slip  that  she  fails  to  borrow  in  column  2  with 
8  over  9;  the  idealized  protocol  inserts  [SD3  .  A  2  ]  prior  to  -  2 


[  SD  3,  A  2,  -2  ,  [A3  ],  -3  ,  [<crossout  ans  3>],  SD  2,  A,,-,  ] 

•  We  do  not  model  this  problem  at  all,  because  of  a  slip  she 
makes  in  doing  a  Borrow-From-Zero  over  the  1  in  column  2 
(a  slip  she  does  not  make  m  the  earlier  problem:  716-  598). 


-■&:  -•v-wytvM 


SB 


COMPARISON  OF  AGENDA  SELECTIONS  ANGELA 


8 


»nd  analyses  for  HiJj 


647 

45  'll  *  2  .  "3 


“  1  *  *  2  i  *  3 


S21  A  ^ ,  D  2 »  *  1  1 


"1  !  *  2  1  *  3  I  *4 


w  v 

23  Sj.Da.A,,.,.-. 


3  S2>  A1f  D2,  -1,  -2,  -3 


136  S2,  A  ,  ,  - ,  ,  D  2,  -  2.  * 


•  w  w 

70  *  1  .  S3  ,  D  3  ,  A  2,  *  2> 


■1  >  S  4,  A2.-2.D4  * 

•  The  idealized  protocol  finishes  answering  columns  3  and  4  There  is  an 
impasse  on  trying  to  decrement  4,  because  it  is  already  decremented.  She 
does  a  Quit  repair.  We  model  it  as  a  Force  repair. 


607  S2  ,  D  2 ,  A  v  - , ,  -  2,  S 4 ,  A  3,  D4  ,  -3  ,  -t 


(S2  ]i  S3  ,  A  1 ,  D  3,  -  1,  -  2,  -  3 


We  count  as  a  slip  her  slash  of  column  2. 


-1  ,  [crossout  arts  ,  and  redo],  [S3  ],  S4  ,  A2  ,  D4  ,  -2  * 

•  The  idealized  protocol  finishes  answering  columns  3  and  4 


V  ' 


Base: 


Ci  >  Ci*, 

Xj>Ci  X--.F.A&D 

Sj  >  Dj 
Aj  >  -  j 


Common: 

'  i  *  Ci+| 

Sj+|  >  Aj 

R-l  >  A  j 

Xj>C,+]  X»F,AS,D 

Standard: 

Di+I  >  A ! 

Weave: 

A|  >  °i+i 
Dj+|  >  -  j 

Slash-Reminds-to  Decrement: 

Aj  >  Dj+) 

*i  >  Di+j 


i 


COMPARISON  OF  AGENDA  SELECTIONS:  HILDA 


'1  »  1  »  2  »  2  i  3 


i ,  A2  ,  SD3  ,  -2  * 


•  Janme  does  not  do  the  -  in  column  3,  since  it  is  just  a  matter  of 
bringing  down  a  0.  The  ideal  protocol  does  the  -  . 


3  A  1 ,  SD  2 ,  A 2  ,  SD  3,  - 1 ,  -  2 , 


T7  Ai,SD2,A2ISD3,A3,SD4,*i,*2,*3  * 

•  This  is  another  instance  of  a  neglected  -  in  the  last  column.  The 
ideal  protocol  does  the  -  . 

2 

9_  A1  ,  ,  [correct  ans1  ],  S9  2,  -2 ,  SD3  * 

•  Another  neglected  -  in  the  last  column.  The  ideal  protocol  does 
the  - 


80  -1 ,  A2  ,  S9  3,  SD4, -2 ,  -  3, 


A,  ,  ,  *  ,  92  ,  -2  .  SD3 ,  -  3  ,  [rewrite  6  over  column  3] 

•  She  inexplicably  rewrites  the  Decrement  over  Top  3  Also,  she  neglects 
to  slash  Top  2  before  writing  9 


wv 

42  *1  *  A  2  >  *21  [^3  *  S3],  SD4,  *3,  -  4 

•  She  does  the  9.  then  the  slash.  The  ideal  protocol  reverses  the  order. 


43 


10012 

214  ^1  »  SD  j  ,  A  j  ,  S9  3,  S9  4,  SO  5  ,  -  ■),  *2,  *3,  *4  ★ 

•  This  is  another  instance  of  a  neglected  -  in  the  last  column.  The  ideal 
protocol  executes  the  - . 

8001 

43  Ai ,  S9  2»  S93,  SD  4 ,  *4,  *3,  ■  2  1  “1 


401 
-  206 


Ai  ,  S9  2>  "i  >  -2  >  SD  3,  -3 


CONSTRAINTS 

Base: 

Xj  >  C,  X-F,  A.SD.-.S9 

A  1  >  -  j 
Ci  >  C  i+i 


Common: 

S9j  >  F, 

SDi+i  >  - 1 
A 1  >  Fj+i 


Prepfirst  (Do  everything  except  -  on  every  column,  then  come  back  and  do 
all  the  -  s  m  order  from  right  to  left): 


~A  i+j  >  -  , 
*F,  >  C, 


- 1  >  -  '+j 

C  >  - 
'SDi  >  Ci 

*  Needed  because  -  no  longer  eliminates  C. 

Needed  because  C  is  processed  before  -  to  its  right  is  done 


Prepfirst/LR  (Same  as  Prepfirst.  but  answers  from  left  to  right): 
Same  constraints  as  Prepfirst,  except: 

■  >  '1  instead  cf  *  >  *1 


Reverse  (Does  A,  thenSD,  then  -  when  processing  Borrow-From): 

-  >  C 
S9j+)  >  -  j 
Fi  >  -  i 

SD  i+j>  -  i  Have  to  do  any  to  the  left  because  of  the  Write  9  Borrow-From-Zero. 

F  i+i  >  ‘  i  Have  to  do  any  to  the  left  because  of  the  Write  9  Borrow-From-Zero. 


Onepass  (Does  the  A  and  the  -  before  proceeding  to  the  Borrow-From): 


-  >  C 

S9i+|  >  - 1 

- 1  >  Fj 

Fi  >  C,  1 

SDi+1  >  C,  / 


Notice  that  this  replaces  the  last  three  in  the  Reverse  set. 
Needed  since  the  Cs  are  no  longer  always  eliminated  by  -  s. 


NOTE  that  none  of  the  foregoing  constraint  sets  handles  problems  7,  9,  and  13, 
which  she  interrupts  her  Borrow-From-Zero  procedure  to  process  a  column. 


COMPARISON  OF  AGENDA  SELECTIONS:  JANINE 


Paul  "stutters"  m  rewriting  his  answer  in  columns  1  and  2. 


*1  i  *2  .  *3  .  *4 


*1  >  *2 i  "3 


S2  »  Ai  ,  -  1 ,  D2  ,  -  2 

^2  '  ^1  >  “ 1 »  D  2  j  *2 

3 2  >  ,  *  1  ,  D2  ,  *  2  1  * 3 

P21  A  •) ,  -  1 ,  D2,  S3,  A  2  ,  D3,  -  2,  S4,  A  3  ,  -  3, 

3  2  >  ^11  "  1 »  ^  2  1  S3,  A  2  ,  -2,  D3>  -  3 

*  2  *  A  1 ,  -  1 ,  D  2 ,  S3,  A  2  ,  -2,  D3,  -  3 ,  -  4 

*2)  A  1  ,  -  1  ,  D2,  *2,  S4,  A3,  D  4,  -  3,  -  4 

'2.  A,,  • , ,  D2,  S5,  D  5,  A4 ,  S4,  D4,  A  3,  S3 

2  >  *3>  '  4>  [*  5  1 

Paul  does  not  write  the  column  5  "0"  in  the  answer  row 


2 


8001 

43 


s*.  D4>  A3 ,  S3,  D3,  A 2.  S3,  D3.  A  1f  - 1(  -2,  *  3»  “4 


r.DNSTRAINTS 


Base: 

Ci  >  Cl+i 

Xi  >  C,  X-F.S.D.  A 

Si  >  Di 
Ai  >  -  i 
Ai  >  S  i 
A  i*i  >  A  i 

Common: 


Si*i  >  A  i 


Siash-Reminds-to  Decrement  (As  m  standard  borrow,  process  the  Borrow-From  goal 

and  its  Slash  subgoal,  but  then  shift  back  to  do  AddiO  and  Diff,  using 
the  slash  mark  to  "remind”  that  the  column  needs  to  be  decremented) 


A  >  Oi*i 

-  ■  >  D.*i 

D 1  >  0 1*  i  i 

C,  >  Oi»i  }  Needed  for  Borrow-From-Zero  (which  he  doesn't  do) 


Weave  (As  m  standard  borrow  process  Borrow-From  goal  and  its  Slash  subgoal  first,  but 
then  shift  back  to  do  the  AddiO.  then  shift  columns  again  to  finish  the 
Borrow-From's  other  subgoal,  Deer,  before  doing  the  Diff): 


A  ,  >  D,*1 

D-*t  >  -  , 

D.*i  >  S, 

Standard  Paul  uses  the  standard  order  when  he  does  a  Borrow-From-Zero. 


Dr*1  >  A  , 

A  i  >  C  ,*  i 


COMPARISON  OF  AGENDA  SELECTIONS:  PAUL 


0 


45 

■1.-2  1-3  [rewrite  ans  2] 

•  He  rewrites  his  answer  to  column  2.  The  model  does  not. 

885 

-  205 

-1 .  -2 .  -3 

83 

-  44 

*2.  A  i,  O2,  -i  ,  *2 

8305 

3 

-1  .  *2  .  *3.  *4 

50 

•  23 

S2 .  A  1,  *i  ,  D2 ,  *2 

562 

3 

S  2  .  A  t  ,  D  2  ,  * .  ,  -  2  .  -  3 

742 

-  136 

-1 .  -2 1  -3 

•  Takes  absolute  differences  instead  of  borrowing. 

106 

•  70 

S3,  A  2,  D2,  S2,  At 

•  Borrows  when  he  shouldn't. 

106 

i -  '«Ve  excluded  these  from 

-  70 

S3  ,  A  2 ,  D  2,  A  ,  - 1 ,  -  2 

•  Borrows  when  he  shouldn't. 

idealized  protocol. 

311 

•  214 

S3  >  D3>  A  2,  s2,  D  2,  At ,  ,  -2  ,  -3 

•  Borrows  from  "Zero '  even  though  it's  a  1 

.%  -  *  -  -JV  a.  1  -  -  *  ’  .  •  _  -  .  «  _V  .  <  -V  -  *  »  -  .  *  . 

49 


6591 

2697 


>2  ,  «  i 


2»  “3i  A  2>  u  3  * 


4» 


1  4  > 


3 »  *  1 »  *  2 i  ‘ 3  -  *4 


887  S2  ,  At  ,  D  2,  S3,  A  2,  D3,  S4  ,  D4,  A3, -2,  *3  * 

•  He  does  not  do  the  final  column  Diff  when  rt's  just  a  zero. 

716 

-  598  ^2>  A  i ,  [S3,  D3,  D2],  A  2>  *i*  *2»  *3 

•  ideal  protocol  substitutes  for  his  sequence  [S  3,  D  3 ,  D2  ]  the  sequence 
(D2.S3.D3]. 


CONSTRAINTS 


Base: 

Ci  >  C,*i 

X,  >  c.  X  =  F.  A.  S.  D 

s,  >  o, 

A,  >  -  1 


Common: 

F.4.1  >  A  1 
S1+1  >  A  1 

A,  >  0,4.1 


Weave 

0 14. 1  >  - 1 

'  >  C.4.! 


Slash-Reminds-to-C-.  crement 
■ .  >  O14.1 

I  >  C,4.1 


Prepfirst-Weave  Does  all  tcp  crocessmg  m  Weave-like  manner  be'ore  writing  anv  answers): 


X  >  ■ 


X  =  c  F  A.  s.  D 


51 


Protocol  and  analyses  for  Robbv 


885 

205 


8305 

3 


83 

44 


907 

607 


106 

70 


6591 

2697 


108 

60 


1236 

497 


1813 

215 


102 

39 


'1  »  “2>  "3 


'  1  »  *  2  >  *  3  »  *  4 


SD  2 >  A  1 1 


’  1  »  ■  2  >  {*3} 

•  He  math-slips  at  column  3:  9-6  =  4. 


-1 ,  A2  ,  SD3 ,  -2  ,  ["correct"  ans  2],  -3 


•  He  "corrects"  his  answer  in  column  2.  The  ideal  protocol  sequence  leaves 
it  alone. 


Ai  ,  SD  2,  [-  2],  A3  ,  SD  4,  -4,  -3 ,  - 


•  He  does  9  -  9  =  1  ideal  protocol  does  Borrow  procedure,  so  for  [-2  ],  the 
ideal  does  [A  2.  •  2  ■  SD3 J. 


*  1  1  A 2 ,  SD3,  -3,  -2 ,  [correct  ans  2] 

•  He  math-slips  at  column  2  answer  and  corrects  ideal  gets  it  right. 


^1,^*2,"l,^2’^*3,”2’^3’"3’*  ’*4 


•  He  does  not  write  his  decrement  in  column  2  or  column  3.  and  fails  to  Borrow 
from  column  4.  ideal  adds  these  steps. 


A-i  !  {'1  }>  *  1  '2  t  SD3,  -3,  -4 

•  Math-slip  at  column  1  Robdy  does  not  write  his  SD  2  or  A2  Ideal  protocol  does. 


A-i  >  {*1  }>  A2  ,  SD  3,  -  2 .  *  3 

•  Math-slid  m  column  1  12-9=4. 


•  '  «  »  H  -  •  • 

’  >  '  •  »  »  .* 


.w 


>Tt] 


•  He  decrements  9  to  7.  Possibly  he  accumulates  decrements,  but  we've  modelled 
his  Borrow-From-Zero  as  a  No-Operation  Borrow-From-Zero,  so  we  don't  catch  this, 
and  decrement  9  to  8, 


A7  ,  ,  *  Dj  ,  *2  i  A3 ,  SD4 ,  -3  ,  -4 

•  He  does  not  write  the  Slash  in  column  2.  ideal  protocol  does. 


27  [S3  *  J,  A1  ,  ,  A  2,  *  21  *  3 

•  He  does  not  write  the  decrement  in  column  3,  Ideal  protocol  does.  Also, 
we  can't  get  his  order,  given  the  No-Operation  Borrow-From-Zero. 

The  Idealizations  at  problems  11  and  13  point  to  problems  with  our  model  of  Robby 
at  the  core  procedure  level,  i.e.  giving  him  a  No-Operatlon  Borrow-From-Zero. 


Cj  >  C  i+1 

X,  >  c, 

A,  >  •  1 


X  =  A,  F,  SO 


Reverse  (When  processing  a  Borrow  goal,  do  A,  then  F,  then  - ): 


A  !  >  F  !+* 

A,  >  SDi*i 
F:*1  >  -  , 

SD1+1  >  -  1 

Onepass  (Do  A  and  -  before  doing  F): 

A,  >  F,+1 
A,  >  SD|+i 

- 1  >  Fi+i 

-1  >  SD1+1 


Standard: 

F  +1  >  A, 
SD1+1  >  A  , 
A,  >  C,+  i 


(NB:  He  only  uses  this  order  on  problem  3) 


COMPARISON  OF  AGENDA  SELECTIONS:  ROBBY 


X,  >  Ci 

Cj  >  cl+, 


X  =  A,  F  ,  9,  SD  (i.e.,  for  the  same  column) 


Common  a 


SO, 


Nailed 


i  >  •  i+i 

1  >  Cj 

'I  >  F.+1 
W  >  F2 

2  >  A1 
►  1  >  A, 


X  *  - 

X  =  A,  F,  9,  SO  (i.e.,  for  any  column) 

}  Note  the  completely  regular  flip-flop  she  does. 


SD  2»  A  i,  -  i ,  -  2 


A 


i  >  *i 


SD2 


*2  »  *3 


A, 


SD  2 1 


‘2  i  "3 


- , ,  [corrects  ans  1  ],  A  2 ,  -2  ,  SD  3  ,  * 

•  She  corrects  her  column  1  answer;  she  does  not  write  0  in  column  3. 


'i  ’  i 


SD2,  SD 


31  ”  2  *  "2 


A, 


‘2  ’ 


SD 


3»  ”  2*  ’ 3 


*1  >  A2  1  A3  ,  SD  3,  -  2,  *  3 


Ai  .  'i  ,  SD2,  *2 1  A3,  -3,  SD4,  -4 

[SD  2],  A  1 ,  *1,  A2,  SD  3 ,  -3  ,  -2 

•  She  does  a  weird  SD  of  0  in  column  2,  which  the  ideal  protocol  does  not  do. 


1  »  A2  ,  A3  ,  SD4  ,  -  4,  -  3,  -  2 


A-t  ,  ,  SD  2 1  A  3,  A  4  ,  SD  3,  A  2 )  *  2 ,  *31  * 4  ,  “5 


43  Ai  .  - 1 »  [s 2 1.  A 3 ,  SD4  ,  [-4.  *2  i  A 2 ],  [write  0  in  column  2] 

•  She  does  a  Slash  in  column  2,  and  a  write  0  over  her  10  at  some  point. 

We  also  did  not  model  the  [-4  .  -  2.  A2]  sequence,  using  instead  (A2 ,  -2  .  - 3,  -4 ], 
after  the  order  in  the  previous  problem,  10012-214. 


CONSTRAINTS 

Base: 

A|  >  - 1 

X,  >  C,  X  =  A.  F,  SO  i tor  same  column) 

C,  >  Ci+i 

Common: 

X,  >  C,  X  =  A.  F,  SO  (for  all  columns) 

■  1  >  C, 

Onepass  (When  processing  a  Borrow  goai,  do  A  and  -  before  doing  the  F): 

A,  >  F,+  i 

- 1  >  Ri*i 

Reverse  (When  processing  a  Borrow  goal,  do  A,  then  F,  then  come 
back  for  the  - ): 

R  '+1  >  •  1 
SO  1+1  >  - 1 
A  1+1  >  ■  1  1 

F  *1  >  - 1  >  Needed  to  handle  her  Borrow-From  -Zero  procedure 

SO  1+]  >  - 1  J 
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